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TECHNICAL  REPORT 

A PRELIMINARY  EXPERIMENT  TO  TEST  INFLUENCES 
ON  HUMAN  UNDERSTANDING  OF  SOFTWARE 

Scientific  advances  in  a particular  discipline  frequently  are  stimulated  by 
work  done  in  some  seemingly  unrelated  discipline.  Recent  work  in  computer  science 
may  provide  just  such  an  advance  to  human  factors  specialists  and  academic  psychol- 
ogists. 

In  1972,  Halstead  first  published  his  software  physics  theory  (later  renamed 
software  science)  stating  that  algorithms  have  measurable  characteristics  analogous 
to  physical  laws.  His  objective  was  to  develop  quantitative  measures  of  the  com- 
plexity of  computer  programs  in  terms  of  language  level,  algorithm  purity,  pro- 
gramming effort,  and  programming  time.  Preliminary  tests  of  the  theory  have  shown 
very  high  correlations  (greater  than  0.90)  bwtween  the  software  physics  metrics  and 
such  dependent  measures  as  the  number  of  bugs  in  programs  (Funami  and  Halstead, 
1975),  programming  time  (Gordon  and  Halstead,  1975),  and  quality  of  programs 
(Halstead,  1973). 

Given  the  huge  individual  differences  in  programming  and  the  inherent  complex- 
ity and  performance  variabilities  inherent  in  such  tasks,  the  high  correlations 
suggest  that  this  theory  warrants  serious  review.  Our  initial  experiment  was 
designed  to  test  understanding  of  computer  programs.  It  asked  the  following 
question: 

"Do  other  independent  variables  allow  us  to  predict  performance  more  accur- 
ately than  Halstead's  theory?" 

This  small  preliminary  experiment  has  been  used  to  validate  materials  and 
procedures  for  a large  experiment  to  test  human  understanding  of  software.  The 
results  of  the  preliminary  test  were  sufficiently  interesting  to  be  reported  at 
this  time.  A larger  experiment  will  be  conducted  with  appropriate  modifications 
as  determined  by  these  results. 
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Current  literature  (Love,  1977;  Shneiderman,  1974  and  1977)  suggests  that 
the  most  sensitive  measure  of  whether  a person  understands  a computer  program  is 
his  ability  to  memorize  the  program  and  reproduce  an  equivalent  program  without 
notes.  It  would  be  extremely  difficult  to  reproduce  a non-trivial  program  with- 
out some  understanding  of  its  function.  In  this  experiment,  the  participants 
were  asked  to  reproduce  programs  and  were  told  that  their  performance  would  be 
evaluated  on  the  functional  correctness  of  the  programs  written. 

Two  independent  variables,  mnemonic  names  and  complexity  of  control  flow, 
were  tested  for  their  relevance  to  understanding  of  computer  programs.  Perfor- 
mance on  the  programming  tasks  was  also  compared  to  the  software  physics  metrics. 

Mnemonic  Complexity 

Three  successively  more  meaningful  levels  of  mnemonic  variable  names  were 
defined  and  tested  for  each  of  three  programs.  Ml  had  randomly  chosen,  single- 
letter variable  names.  M2  had  variable  names  from  1 to  4 characters  in  length, 
selected  to  be  meaningful,  but  not  as  mnemonic  as  M3,  the  most  mnemonic  level. 

For  M3,  the  variable  names  were  chosen  to  be  as  meaningful  as  possible  and  were 
4 to  6 characters  in  length.  (A  maximum  of  6 characters  is  allowed  by  most 
standard  FORTRAN  compilers.)  No  variable  had  exactly  the  same  name  in  any  two 
mnemonic  versions  (Table  1). 

Control  Flow  Complexity 

Three  levels  of  complexity  of  control  flow  were  also  defined.  S3  was  the 
cleanest  control  flow,  following  accepted  structured  programming  rules;  no  three- 
way  transfer  of  control  statements  and  no  backward  transfer  of  control . S2  and  SI 
were  successively  less  straightforward.  In  S2,  minor  violations  of  the  "top-to- 
bottom"  control  flow  sequence  were  allowed.  At  the  SI,  or  least  structured  level, 
there  was  more  frequent  use  of  the  GO  TO  statement  and  backward  transfers  of 
control  were  allowed.  The  three-way  transfer  of  control  statement  [IF  ( )-,o,+] 
was  allowed  only  at  the  SI  level  (Table  2). 

Halstead's  E 


Halstead's  theory  of  software  science  presents  a method  for  measuring  the 
complexity  of  software.  E,  the  amount  of  "effort"  required  to  generate  a program, 
can  be  calculated  from  simple  counts  taken  from  the  program  itself.  The  calculations 
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Table  1 . Mnemonic  Variable  Names 


PROGRAM  PI 

PROGRAM  P2 

PROGRAM 

P3 

Ml 

M2 

M3 

Ml 

M2 

M3 

Ml 

M2 

M3 

D 

A 

ASCORE 

P 

X 

XVALUE 

L 

N 

NUMBER 

E 

S 

VECTOR 

Q 

Y 

YVALUE 

B 

X 

XVALUE 

F 

T 

TOTAL 

R 

F 

FTABLE 

C 

Y 

.'VALUE 

B 

AV 

AVERAGE 

M 

NC 

NUMCOL 

E 

XCOR 

XCOORD 

R 

SD 

STDEV 

N 

NR 

NUMROU 

0 

PL 

PLOT 

P 

VMIN 

VECMIN 

V 

XB 

XBAR 

F 

YCOR 

YCOORD 

Q 

VMAX 

VECMAX 

W 

YB 

YBAR 

G 

XCO 

XCOOR 

L 

NV 

NUMVAR 

A 

FAC 

FACTOR 

H 

FAC 

FACTOR 

M 

NO 

NUMOBS 

B 

XJI 

YCORRl 

P 

RBOT 

RMINI 

V 

SCNT 

COUNT 

C 

XJ 

YC0RR2 

Q 

RTOP 

RMAXI 

D 

ANS 

ANSWER 

R 

XBOT 

XMINI 

0 

NO 

NUMPTS 

S 

XTOP 

XMAXI 

T 

RNG 

XRANGE 

Ml 

* RANDOMLY  CHOSEN,  SINGLE  LETTER 

M2 

= 1-4 

MEANINGFUL  LETTERS 

U 

SCL 

XSCALE 

M3 

= 4-6 

MEANINGFUL  LETTERS 

V 

CNS 

XCONST 

W 

STP 

STEP 

M 

II 

IMINNS 

I 

IVAL 

IVALUE 

are  based  on  the  two  quantities:  1)  the  sum  of  the  number  of  distinct  operators 

and  operands,  and  2)  the  sum  of  the  total  number  of  operators  and  operands.  From 
these  sums,  Halstead  derives  the  number  of  mental  comparisons  required  to  generate 
a program.  (A  primary  assumption  is  that  human  beings  use  a search  technique  as 
efficient  as  a binary  search  to  retrieve  the  components  of  each  program  instruction. 
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Table  2.  Control  Structures  Allowed  in  the  Three  Levels  of  Complexity 


MOST  STRUCTURED 
(S3) 


PARTIALLY  STRUCTURED 
(S2) 


LEAST  STRUCTURED 
(SI) 


(no  exits  from  loops) 


DO,  exits  allowed 


IF  ( ) assignment 

IF  ( ) GO  TO  


IF  ( ) - . 0 , + 


GO  TO 

(never  backward) 


GO  TO  GO  TO 

(limited  returns  above)  (no  constraints) 


Since  different  programming  languages  produce  widely  varying  numbers  of 
instructions,  the  number  of  elementary  mental  discriminations  for  each  mental 
comparison  varies  with  the  language  used.  When  a correction  is  made  to  account 
for  these  differences,  one  can  define  E in  terms  of  the  number  of  mental  discrim- 
inations/program: 


c _ No.  of  Comparisons 
Program 


Average  No.  of  Discriminations 
Compari son 


A thorough  discussion  of  the  theory  and  calculations  are  beyond  the  scope  of 
this  paper.  See  Fitzsinmons  and  Love  (1976)  or  Halstead  (1977)  for  more  details. 

Testing  Procedure 

Eight  participants  were  tested  in  a group.  Seven  of  them  had  had  at  least 
one  year  of  prograirming  experience  in  FORTRAN.  The  eighth  individual  was  an 
experienced  COBOL  programmer  with  minimal  knowledge  of  FORTRAN. 

Instructions  were  given  to  participants  orally  (Appendix  A).  Each  individual 
received  a program,  studied  it  for  25  minutes,  and  then  attempted  to  reconstruct 
it  from  memory  in  the  next  20  minutes.  Three  such  tasks  were  given  to  the  partici- 
pants, with  a 20-minute  rest  period  between  the  last  two  tasks. 

Each  individual  saw  each  of  the  three  programs  once,  each  level  of  mnemonics 
once,  and  each  level  of  control  flow  once.  The  experimental  design  (Kirk,  1968^ 
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and  Mendenhall,  1968)  was  a fractional  split-plot  factorial  (Table  3).  All 
participants  received  a least  structured  (or  most  difficult  in  terms  of  control 
flow)  program  first.  Secondly,  each  received  a very  structured  program.  Lastly, 
each  received  a program  with  intermediate  structure. 

Table  3.  Experimental  Design  for  the  ONR  Pretest 


SUBJECTS 


ORDER  OF  OCCURRENCE 


2 


1. 

4, 

7 

PI 

M3 

SI 

P2 

M2 

S3 

P3 

Ml 

S2 

2, 

5, 

3 

P3 

M2 

SI 

PI 

Ml 

S3 

P2 

M3 

S2 

3, 

6 

P2 

Ml 

SI 

P3 

M3 

S3 

PI 

M2 

S2 

Program  Number 

Least  Structured 
Moderately  Structured 
Most  Structured 

Least  Mnemonic 
Moderately  Mnemonic 
Most  Mnemonic 


Informal  interviews  with  the  participants  following  the  experiment  were  con- 
ducted to  determine  their  cotments  on  the  experimental  materials  and  techniques 
used. 

Programs 

Choice  of  program  is  a significant  factor  in  any  programming  test.  Previous 
experience  with  a particular  type  of  program  aids  a programmer's  understanding 
greatly.  Since  it  would  be  difficult  to  find  program  functions  with  which  none 
of  the  participants  had  any  previous  experience,  it  was  decided  to  use  common 
applications  which  were  readily  understandable  to  most  programmers. 

The  programs  chosen  are  described  in  Table  4.  Each  was  preceded  by  a short 
description  of  its  function  (5  to  10  lines).  No  comments  were  intersperced  in 
the  code,  and  no  indenting  of  the  code  was  done.  All  declaration  statements 
appeared  at  the  beginning,  and  all  FORMATS  were  at  the  end. 
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Table  4.  Statistics  for  the  Three  Programs  Tested 


PROGRAM  # 

PI 

P2 

P3 

PURPOSE  OF  PROGRAM 

CALCULATE  MEANS, 
MAXIMUMS,  MINIMUMS 
AND  STANDARD 
DEVIATIONS  FOR  A 
SET  OF  DATA 

PERFORM 

two-dimensional 

INTERPOLATION  FOR 
A POINT,  GIVEN  A 
TABLE  OF  VALUES 

scale  a set  of 

COORDINATES  (X,Y) 
AND  PRINT  A 
SCATTER  PLOT 

STRUCTURE  LEVEL 

SI 

S2 

S3 

SI 

S2 

S3 

SI 

S2 

S3 

# STATEMENTS  IN  PROGRAM 

46 

36 

32 

56 

50 

34 

83 

79 

76 

# ASSIGNMENT  ST. 

24 

20 

16 

21 

17 

14 

33 

35 

34 

# CONTROL  FLOW  ST. 

20 

14 

14 

23 

22 

12 

40 

34 

32 

TOTAL  # IF  STATEMENTS 

7 

5 

5 

8 

6 

3 

4 

4 

3 

# GO  TO  STATEMENTS 

7 

1 

1 

8 

5 

2 

9 

2 

1 

HALSTEAD'S  E 
(thousands  of  mental 
discriminations) 

121 

61 

59 

108 

54 

40 

142 

1 19 

80 

Different  versions  of  a program  varied  in  length.  Programs  1 and  2 averaged  i 

38  and  47  statements  respectively,  while  Program  3,  which  included  a subroutine,  i 

averaged  74  statements.  » 

• 

Scoring  | 

All  of  the  programs  were  scored  by  the  same  grader.  The  criterion  for  scoring  | 

each  statement  (or  each  related  group  of  statements)  was  functional  correctness. 

■ 

Variable  names  and  statement  numbers  different  from  those  in  the  original  program  | 

were  counted  correct  when  used  consistently.  Variations  of  control  structures  < 

i 

different  from  the  original  were  noted,  but  allowed  if  correct  functionally. 

The  errors  in  the  programs  were  divided  into  four  categories  by  statement 
type:  1)  assignment,  2)  control  flow,  3)  other,  and  4)  extraneous  (extra  state- 

ments the  individual  added  that  were  unnecessary).  Table  5 shows  the  categoriz- 
ation of  statements  in  the  original  programs. 

The  individual  errors  within  statements  were  also  counted,  and  they  fell  into 
three  general  categories:  syntax  errors,  assignment  errors,  and  transfer  of 

control  errors. 
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RESULTS 

There  were  not  enough  errors  to  do  an  analysis  by  error  type.  Therefore,  the 
analyses  concentrated  on  four  dependent  variables,  relating  to  statements  only: 

PERCENT  OF  STATEMENTS  ATTEMPTED 
PERCENT  OF  STATEMENTS  CORRECT 
PERCENT  OF  ASSIGNMENT  STATEMENTS  CORRECT 
PERCENT  OF  CONTROL  FLOW  STATEMENTS  CORRECT 

The  overall  mean  of  the  percentage  of  statements  done  correctly  was  50.3, 
indicating  that  the  difficulty  of  the  task  assigned  was  neither  too  difficult  nor 
trivial . 

Analyses  of  Variance 

Analyses  of  variance  for  each  of  the  dependent  variables  (Nie,  1975)  showed 
no  significant  effects  caused  by  mnemonic  level,  although  it  appeared  that  there 
might  be  a trend  to  do  slightly  better  with  the  single-letter,  randomly  assigned 
variable  names  (Figure  1).  This  is  not  the  result  one  would  expect  here,  since 
longer,  meaningful  names  are  intuitively  easier  to  understand.  Further  work  might 
investigate  more  levels  of  meaningful  ness  to  try  to  explain  this  phenomenon. 

Clearly,  a more  straightforward  control  path  should  produce  a more  under- 
standable program.  As  expected,  significant  differences  were  produced  among 
levels  of  control  flow  for  all  four  dependent  variables. 

This  result  must  be  viewed  with  some  retraint,  however,  because  for  each 
program  the  total  number  of  statements  decreased  as  the  control  flow  became  easier, 
thus  making  it  possible  to  complete  a greater  percentage  of  the  total  statements 
while  not  actually  completing  many  more  statements.  Further,  one  participant  did 
not  attempt  to  complete  any  statements  on  the  least  structured  version  of  the 
longest  program,  P3.  The  percent  of  statements  correct  on  the  other  two  programs 
was  74  and  81,  indicating  that  Control  Flow  Level  1 values  in  Figure  1 would  have 
been  increased  if  the  individual  had  completed  any  statements  correctly. 

One  particularly  interesting  result  (Figure  1)  is  that  the  percentages  of 
control  flow  statements  done  correctly  are  markedly  less  than  the  percentages  of 
assignment  statements  done  correctly  in  each  case.  This  suggests  that  control 
flow  statements  are  more  difficult  to  reproduce  than  assignment  statements,  for 
some  as  yet  unknown  reason. 
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S STATBCWS  ATTB*TEO 
X STATEMENTS  COIWECT 


X CONTROL  aOU  STATEMENTS  CONNECT 
X ASSIGNMENT  STATEMENTS  CONNECT 
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Figure  I.  Analysis  of  Variance  Results  for  Three  Mnemonic  Levels 
and  Three  Levels  of  Control  Flow 


Regression  Analysis 


A multiple  regression  analysis  (Kerlinger,  1973)  was  done,  using  effect 
coding  for  the  independent  categorical  variables,  participants,  program,  level  of 
control,  and  level  of  mnemonic  variable  names.  The  particular  split-plot  factorial 
design  used  in  this  experiment  had  the  order  of  presentation  confounded  with  level 
of  control  flow,  making  it  impossible  to  assess  these  variables  separately.  The 
experiment  was  designed  under  the  false  assumption  that  affects  due  to  control  flow 
and  order  of  presentation  were  monotonic  and  linear.  These  affects  will  be 
separated  in  future  experiments. 


Table  6 presents  the  change  in  R Square  due  to  the  independent  variables  and 
the  significance  of  the  regression  equation  as  each  variable  was  included.  A total 
of  92  percent  of  the  variance  accounted  for  is  extremely  respectable. 
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Table  6.  Results  of  Multiple  Step-Wise  Regression  Analysis 


ORDER  OF 
ENTRY  INTO 
REGRESSION 
EQUATION 

VARIABLE 

PERCENT  CHANGE 
IN 

R SQUARE 

SIGNIFICANCE 

OF 

REGRESSION 

EQUATION 

1 

PROGRAM 

43 

0.003 

2 

SUBJECT 

18 

0.065 

3 

CONTROL  FLOW/ 

ORDER  OF  PRESENTATION 

26 

0.001 

3 

MNEMONIC  LEVEL 

5 

0.001 

4 

INTERACTION  CXM 

0 

* 

TOTAL 

92 

Programs  accounted  for  the  largest  amount  of  change  in  R Square  (43  percent). 
Comparing  the  total  number  of  statements  per  program  to  the  percentage  of  state- 
ments done  correctly,  the  correlation  coefficient  was  found  to  be  -0.70.  Program 
P3,  which  averaged  74  statements,  as  opposed  to  38  and  47  for  PI  and  P2  respec- 
tively, largely  accounted  for  the  decrease  in  the  number  of  statements  correctly 
done. 

For  each  of  the  programs  used,  a measure  of  psychological  complexity  was 
computed,  using  a current  theory  of  software  science.  A measure  of  the  mental 
effort  required  to  create  a program,  described  by  Professor  Halstead  of  Purdue 
University  (Halstead,  1975),  had  previously  been  shown  to  be  an  excellent  predictor 
of  1)  the  number  of  errors  in  a program,  and  2)  the  time  required  to  implement  a 
program. 

Halstead's  E for  each  of  the  programs  used  is  shown  in  Table  4.  It  was  found 
to  be  correlated  -0,81  with  the  precentage  of  statements  correctly  recalled 
(Figure  2).  Entering  only  E into  the  regression  equation  (Table  7),  it  accounted 
•:  for  65  percent  of  the  variance. 

* This  interaction  could  not  be  evaluated  with  the  experimental  design  used  and  was, 
..  therefore,  not  entered  into  the  regression  equation. 
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(2  Indicates  two  data  points 
at  the  same  location) 


R*-  0.8 


I*  Pearson  Correlation  Coefficient  I 

I I I I I I I ' I 1 * I 

30  42  54  66  78  90  102  114  126  138  ISO 

HALSTEAD'S  E (Thousands  of  Mental  Discriminations) 

Figure  2.  A Comparison  of  Halstead's  E and  the  Percent  of  Statements 
Correctly  Completed  (R  * 0.8)  for  Each  Program 

Table  7.  Results  of  Multiple  Step-Wise  Regression  Analysis  with  Halstead's  E 


VARIABLE 


PERCENT  CHANGE 
IN 

R SQUARE 


SIGNIFICANCE 

OF 

REGRESSION  EQUATION 
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Comparing  Tables  6 and  7,  we  can  recognize  the  power  of  Halstead's  E as  a 
^ predictor.  In  Table  6,  programs  and  control  flow  accounted  for  69  percent  of  the 

^ variance.  In  Table  7,  Halstead's  E,  programs,  and  control  flow  accounted  for 

68  percent  of  the  variance  with  programs  and  control  flow  responsible  for  only 
^ 3 percent.  Clearly,  we  would  improve  future  experiments  with  a more  uniform 

f distribution  of  the  E factor.  The  distribution  in  Figure  2 appears  to  be  skewed 

j and  bimodal,  whereas  in  future  experiments  the  range  of  program  materials  will 

permit  E to  assume  a more  continuous  distribution. 
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Only  24  observations  were  obtained  in  this  incomplete,  split-plot  factorial 
design,  and  it  is  not  appropriate  to  draw  far-reaching  conclusions  on  the  basis  of 
this  work.  However,  several  findings  do  merit  discussion. 

When  evaluating  levels  of  mnemonic  variable  names,  we  expected  longer,  more 
meaningful  names  to  make  programs  easier  to  understand.  However,  no  significant 
differences  were  found  among  the  levels  of  variable  names.  Contrary  to  intuition, 
the  single,  randomly  chosen  letters  resulted  in  better  (albeit  nonsignificant) 
recall  than  either  1 to  4 or  4 to  6 "meaningful"  letters.  Further  work  must  eval- 
uate variable  names  more  comprehensively. 

Control  flow  made  a large  difference  in  performance,  as  expected.  The  more 
structured  programs  were  recalled  significantly  better  than  the  less  structured 
ones.  In  addition,  the  percentage  of  control  flow  statememts  correctly  completed 
across  all  conditions  was  dramatically  less  than  the  percentage  of  assignment 
statements  correctly  completed,  indicating  that  control  flow  statements  themselves 
are  more  difficult  to  understand. 

Halstead's  E,  a measure  of  the  effort  required  to  complete  a program,  was 
found  to  be  highly  correlated  (-0.81)  with  the  percent  of  statements  correctly 
completed.  A correlation  of  this  magnitude  is  impressive  in  an  experiment 
designed  to  test  counterintuitive  theoretical  predictions. 

We  recognize  that  the  effects  of  the  independent  variables,  complexity  of 
control  flow,  and  mnemonic  variable  names  cannot  be  unambiguously  determined  as 
a result  of  this  experiment.  Control  flow  appears  to  be  highly  significant,  yet 
possible  effects  due  to  the  confounding  with  order  of  presentation  cannot  be 
ruled  out.  Further  work  is  needed  to  determine  the  effects  of  variable  names  in 
a more  sensitive  study. 
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SCRIPT  FOR  TESTING 
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Hi! 

I'm , and  I'm  with  General  Electric's  Information 

Systems  Programs  in  Arlington,  Virginia.  Our  group  is  interested  in  experimenting 
with  different  types  of  programs  to  see  how  we  can  make  a programmer's  job  easier. 
Our  work  is  being  funded  by  the  Office  of  Naval  Research,  and  we  are  paying  for 
your  services  from  that  contract. 

Let  me  assure  you  that  we  are  evaluating  alternative  programming  techniques 
and  not  evaluating  programmers.  The  experiment  was  designed  so  that  you  serve  as 
your  own  control.  Your  performance  on  a program  will  be  compared  only  to  your 
performance  on  other  programs.  Your  only  competition  is  yourself.  As  a result, 
it  is  not  necessary  for  you  to  put  your  names  on  your  papers. 

If  you  look  over  at  your  neighbor  during  the  test,  you  will  see  that  each  of 
you  is  doing  a different  chore.  If  someone  else  finishes  earlier  than  you,  don't 
be  concerned  about  it.  He  or  she  will  have  been  working  on  something  else,  which 
might  not  take  as  much  time  as  your  chore. 

We  are  going  to  begin  this  morning  by  asking  you  to  study  a FORTRAN  program 
for  25  minutes.  You  may  write  anything,  draw  flow  charts,  or  whatever  helps  you 
to  understand  the  program.  When  the  25  minutes  are  up.  I'll  ask  you  to  hand  in 
the  programs  and  all  of  your  notes.  We'll  then  give  you  20  minutes  to  reconstruct 
the  program  from  memory  as  closely  as  possible  to  the  original. 

On  some  of  these  programs,  you  may  think  of  a better  way  to  rewrite  it,  but 
I'm  asking  you  to  restrain  your  creativity  and  just  try  to  reproduce  the  original 
program  as  accurately  as  possible,  line  by  line.  That  means  using  the  same 
constructions.  If  we  have  a subroutine,  please  use  a subroutine.  If  we  have  a DO 
loop,  please  do  it  with  a DO  loop.  However,  it's  not  necessary  to  memorize  state- 
ment numbers.  If  your  statements  are  numbered  differently  from  mine,  it's  O.K., 
as  long  as  the  program  still  does  the  same  job.  If  you  can  use  the  same  variable 
names,  that  will  help  me  to  score  the  programs,  but  it  will  be  O.K.  if  you  can't 
do  that. 

Does  anyone  have  any  questions? 
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